This would give you a pass/fail test for a certain situation (the one you perform the test in), but does not tell you much about the riser itself, nor how it will work in other situations. If you know and can the environment in which the riser will be used, this testing is suitable and a lot cheaper than full signal testing.just thinking out loud: what about a really simple 'terminator-like' device and a driver? the terminator comes instead of a real pcie device.
e.g. the terminator could be something looping back data on each lane (or across lanes, etc). it must be an active component because it needs to negotiate the pcie version, number of lanes used, TLP length, etc). passive loopback would be too easy
so a pcie device with an fpga would do it. the problem is that they are really expensive so killing the whole idea of 'cheap' testing. altera and xilinx might have some nice boards.
unfortunately a cheap(er) x1 device is not an option because most likely we can not say that if one lane has a good quality then all the lanes have and regarding signal integrity this is absolutely an invalid test for multi-lane risers.
Also, having a NIC with enough bandwidth is really expensive, too. a 2x100G NIC doing a loopback would be pretty close to what I am thinking about. (either having a loopback cable or configured for doing loopback on chip level). the problem is not only the expensive NIC but also the CPU required for generating and capturing this amount of data. trafgen could be something like https://trex-tgn.cisco.com/ or http://dpdk.org/browse/apps/pktgen-dpdk/refs/
another option could be to add this feature to an existing videocard driver and write a tool like the Furmark. Or using Furmark itself if it can generate really high traffic with some settings on pcie. Intel's PCM (performance count monitor) or Microsoft's Xperf might give some information about the occupancy of the lanes but I am unaware of any counters like retransmit or so.
The benefit to proper 'eye pattern' testing is it tells you not only whether the riser will work, but exactly how much margin you have before it will get out of spec, and how it will go out of spec (which lets you track down which element is failing and how to improve it). For example, you can easily tell the difference between a crosstalk issue (and use the timing to determine exactly where in the riser the crosstalk is occurring) from an impedance mismatch issue. It also gives direct feedback on what's happening during interference testing, so you can track down where signal is 'leaking'.
