When deleting a WebPart from a page with PowerShell you need to get the WebPartManager from that page and call the Delete() method.

There is one thing you have to keep in mind in case of pages in libraries with content approval. Here, it is important you check-out the page before getting the WebPartManager for the page. Otherwise PowerShell will complain that the page is checked out by someone else (even though it’s really you who checked out the page).