If an order placed by you is rejected, the reason behind the rejection can be checked in Order Book. The reasons can range from insufficient margin, incorrect order inputs, scrips not available for trading etc.
Yes, the order will get rejected if you do not have sufficient funds in your account. Hence it is important for you to maintain the required balance in your bank account or IIFL Securities Ledger.