Solr4 To Solr6 CPU load issues

I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but overall the CPU load on solr6.6 is double the load on solr4.0 and hence I am not able to roll solr6.6 to 100% of my traffic. 

Some Key Stats In Performance of Sol6 Vs Solr4
Document cache usage increased from .98 from .14 
Query Result cache usage increased from .10 from .24 
Filter cache same as .94 
Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field multivalued concept was changed from solr4 to solr6) 

Please Help Note: I have given document cache 3 times memory for doc.cache.