I do not think I have ever experienced an issue like this. Do you mean the same playlist still plays the deleted songs? Or do you just find them elsewhere like in Flow or in your listening queue?
Maybe some screenshots can help explain what is going on.

@Søren Olsen 

If a playlist is running and you remove individual songs, those removed songs will still be in the queue and will continue to be played. If after editing the playlist the flow is started briefly (i.e. the queue is reset) and then the edited playlist is started again, the removed songs should no longer be played or should not appear in the queue.

Otherwise, screenshots or a screenvideo are actually quite helpful, as @Marcormark already wrote.

Thanks for all of your input. Much appreciated. 

Problem was probably that the songs were still in the queue. I have now re-organized the playlist and saved it. This also removed the deleted songs. So all is good. :-)