A portrait by Amedeo Modigliani sold for less than expected and several artworks went unsold as China’s best-known art investor liquidated part of his collection.