JEE Main 2020 — Hyperbola Question with Solution

A line $y=mx+c$ is a tangent to the hyperbola $\frac{x^2}{a^2}-\frac{y^2}{b^2}=1$ if $c^2=a^2{m}^2-b^2$ and the line is tangent to the circle $x^2+y^2=r^2$ is $c^2=r^2\left(1+m^2\right).$

Hence, for the given line and the hyperbola we have, $c^2=100m^2-64 ...\left(i\right)$ and for the line and the circle, we have 

$c^2=36\left(1+m^2\right) ...\left(ii\right)$

On comparing the above two equations, we get

$100m^2-64=36+36m^2$

$64m^2=100$

$m^2=\frac{100}{64}$ $\Rightarrow m=\frac{10}{8}=\frac{5}{4}.$

$\Rightarrow c^2=36\left(1+\frac{25}{16}\right)=\frac{369}{4}.$