ResponseRewrite problem

im working on a website (lets call it domain.com for ease) i have setup an error handler (.ashx file) and it is working perfectly, the result i want to achieve is something similar to this:

1. the user access the website requiring a file not present, say www.domain.com/A0S5-D5S8.txt
2. the web.config catches the 404 error and redirect to the error handler, say handler.ashx
3. the handler read the requested URL and answer accordingly to my BL layer

the issue comes when i try to use proper web.config (see below) with a .ASHX file rather than a .ASPX no clue why

<customErrors defaultRedirect="~/index.aspx" redirectMode="ResponseRewrite" mode="On"><error statusCode="404" redirect="~/handler.ashx" /></customErrors>