Here are my thoughts on gearing and tire size. The 6L80E has much better gearing compared to the stock JK trans. 1st gear is much lower and it has 2 overdrives. Offroad you will be fine with the stock 241J transfer case and whatever gear you choose. The super low 1st gear will give you all you need in 4 low. Highway driving is what you need to be thinking about.
My personal Jeep 2011 JKU 6.2 around 500HP and 6L80E. Bumpers, winch all the usual stuff. 37" MTRs and 4.56 gears.
With 4.56 gears and 37's it does great on the highway and runs around 1900 rpm @ 70MPH it will hold 6th gear and chug over most hills without even downshifting. Granted I have more power than most but I have ran 5.3s and stock 6.2s in this jeep with the same gearing and tire size and it always did well.
We just finished up a 10JKU 6.2 430HP 37's 4.56 gears. It did very well and would run 80 MPH all day long without breaking a sweat or dropping out of 6th gear.
A 5.3 in a big heavy JK may struggle a little bit holding 6th gear as it doesn't have as much torque as the 6.2 but with proper transmission programing it will drop to 5th, pick up about 400 rpm's and pull over most hills without straining.
We usually like to run the 5.3's with a little more gear if you don't have access to a good transmission tuner.
Transmission tuning is key here guys. The calibration that is in the trans computer is set up for the factory vehicle it was pulled from. Everything they do is geared towards a stock vehicle with light tires and crappy gearing. All they care about is smooth shifts, fuel economy and preserving the transmission from warranty claims. If you just drop in the trans without tuning it will drive and shift no problem. But it won't be anywhere near calibrated for the big heavy Jeep, tires size or gear ratio that your running. They will usually shift up too fast, downshift way to aggressively and be general mushy. We have spent a lot of time and effort making real world calibrations for these swaps and the improvements are truly night and day.
Fuel mileage was our first goal and by calibrating the transmission we were able to see pretty significant gains.
Second was highway manners. You have plenty of power now so lets use it. The stock calibration would kick out of 6th way to easy. we rework the shift schedule and allow you to use the torque of the engine to float over hills and maintain speeds without it kicking down and pulling high rpm's. Again this gained us fuel mileage and made it a pleasure to drive.
Third we worked on shift feel. We reworked the shift times and pressure onsets. This allows us to "tune" the trans feel to the weight of the vehicle. We can make if have a firm shift or an all out bark the tires "shift kit" feel. Personal preference here guys, you want bark the tires and snap necks? it can be done. We can even base the shift pressure on throttle position so you can have a nice smooth shift at light throttle and a hard aggressive shift at wide open throttle.
Forth we work on the shift schedule and shift points. This allows us to calibrate when and where it will shift. We allow it to build a little more speed before it shifts and make the lower speed downshifts a little more aggressive. While this may hurt fuel economy a bit it makes the Jeep feel like it has more power and is much more fun to drive.
Fifth we work on the TOW Haul mode. When in tow haul mode you have a separate shift schedule and we can tailer it to whatever you want. If you pull a camper or boat, we will set it up for max towing. If you want to play race car driver on the backroads we can do that too. Personally I have mine set to race car. It will REV match downshift coming into the corners like a Corvette and sounds freaking awesome.
Sixth we set up the 4WD Low shift pattern. This allows us to make a dedicated offroad shift schedule. This is useful when offroad and would be tailored to how you wheel. We even can add in a 2nd gear start for slick conditions like mud, sand and snow.
I crunched the numbers to show you guys what tire size and gear ratio would be with a 6L80E. These are all a 70MPH and probably within 25rpms depending on how good my math is.
6L08E....4.56...................4.88........................5.13..................RPM @ 70MPH
.......6th.........5th........6th........5th...........6th.........5th
35"..2050......2600......2200.....2800........2300.......2950
37"..1950......2450......2100.....2650........2200.......2800
40"..1800......2300......1925.....2450........2000.......2600
Remember the faster you spin the engine the more heat it makes and fuel mileage will go down. If you spin it too slow you will be under the power curve and constantly having to downshift which will suck to drive and will suck fuel also.
I feel that the sweet spot for 5.3's is between 1900 and 2200 rpm's
The sweet spot for 6.2's is 1700 and 2000 rpm's
The sweet spot for 6.0's is 1600 and 1900 rpm's
Of coarse all of this will depend on your Jeep and driving style. I light 2 door that drives primary on the highway won't need to spin the engine near as fast as a big heavy 4 door.
If you don't spend a lot of time on the highway you can get away with running more gear.