This is normal. Many things (libraries, devices, fonts etc) stay in memory and the memory is only reclaimed once needed.
Obviously there also are memory leaks in applications, too.
If you really must you can try reclaiming memory manually with
avail flush