You're not the first to think of such a thing. Personally, when Dr. Michal Schultz gets AROS working on the RasPi 2 and 3, that may be a better solution because Linux runs heavily on 1 GiB shared memory systems. I replaced my RasPi with an ODroid XU4 until something lighter comes along.