If you're just experimenting just do everyone a favor and use RJ45 or screw terminals. These special connectors are really expensive and require special crimp tools.
RJ45 needs to die. As does every other connector that was ever used for more than one incompatible thing. USB got it right: build a custom connector for your application so that nobody will ever plug the wring thing in.
I don't care what you replace Ethernet RJ45 with, but make it a standard that is only used for Ethernet.
Weird connectors are hard to hack on. You bring up USB but I couldn't think of a worse example. USB got it wrong, it supports so many different things on a single connector that in some situations it can be impossible to determine what will happen when you connect device A to device B with cable C. Now don't get me wrong, it's great for plugging things into your laptop, but the world of cabling is so much bigger than that, and we don't really need to bring in even more BS connectors where they don't belong.
At the end of the day it's someone's job to actually run cable, terminate and test it, and attach equipment to it that people are responsible for. You have to remember that the physical cable plant is infrastructure itself whether it's connected to live equipment or not. There's already a big enough problem with people trying to do things like route display cables through buildings, for example - the workaround that I'm seeing a lot is that people are using specialized signal converters that send HDMI/DisplayPort over a pair of Cat5e/Cat6 cables, just so that structured cabling can use normal cabling/networking supplies (i.e. keystone jacks, patch panels, conduit sizes, testing/tools, etc.) without wasting so much time with planning for projectors/displays ahead of time etc.
RJ45 is for twisted pair cabling. It's not specifically for Ethernet. 802.3cg only uses 1 pair. If you use the center pair then it works for TIA-568A, TIA-568B, or USOC. It's great for experimenting since everyone already has twisted pair patch cables to play with. Barring that, just give me screws that I can put the wires into so I don't need to think about connectors at all.
I did HDMI over Ethernet at home. It works with a 19 m long Cat 6 cable plus two 3 m patch cables. Beware of bending the cable too much if it must go inside pipes in the walls.
Whatever connector you stick on the end, as long as it gains popularity someone will use it in weird ways. The second your start selling these connectors widely and for affordable prices, they're going to get reused.
See: USB over D-SUB, serial over RJ45, the wide range of proprietary protocols that cheap IDE cables were (and probably still are) use for today, and weird edge cases like reusing the sturdy DMX ports to power sex toys.
8P8C is fine for ethernet, it doesn't need replacing. Nobody uses it for telephony modems anymore and very rarely will an average user encounter a compatible plug that wasn't made for ethernet.
I'd rather see fringe use cases like serial over RJ45 use a better connector than to have to cut off, strip, and wire up a new connector for every new network appliance I'll buy in the next 10 years.
To be extra pedantic, it's serial with an "8P8C modular connector", not over RJ45. Sure, the physical connector is the same thing, but the RJ- standards define the uses of the various pins, not just the physical connector. RJ45, RJ49, and RJ61 all use the same "8P8C modular connector". RJ45S is for only one data line, with a programming resistor. RJ49C is an 8P8C carrying ISDN via NT1, RJ61 is four telephone lines on an 8P8C. There's no RJ standard for RS232 or for Ethernet at all!
Everyone (including me) still calls it RJ45 because "8P8C modular connector" is far too long a name.
It's purely an analog device. Presumably they used USB because the cables are cheap, shielded, and have enough conductors for what they needed. (Probably ground plus signal wires for two linear pots and a pressure sensor.)
I hope they designed it so it doesn't fry something if you connect it to a real usb device, but I'm not inclined to try it.