You won't find anything smaller than ITX for a s1150 CPU; mini STX is socket 1151 (Skylake) and newer.
Thin ITX boards will save you vertical height, but have a 1x (rarely a 4x) PCIe slot, if at all. None of this is going to be helpful for GPU bandwidth, even on a GT1030.
Any H81/B85/H87 ITX board will be fine with the i5, you will only need a Z87Z97 if it's a
K CPU that you can overclock.
If you want to make it smaller, you could use a specialist PSU (e.g. Laptop brick + 'Pico PSU' style DC converter) smaller than an SFX / Flex ATX / TFX PSU, but it will cost more.
An
Asus Q87T would do the trick with PCIe 4x slot and a suitable brick. You will need SODIMM Laptop RAM for this like all thin ITX options, and it may be expensive (ones on ebay UK are for example)