Using styled components "as" prop with typescript

I'm currently building a pattern library in which I've built a Button component using React and styled-components. Based on the Button component, I want all my Links component to look exactly the same and receive exactly the same props. For that purpose, I'm using the as prop from styled-components, which allows me to use that already built element as another tag or component.

Button Component

import * as React from 'react'
import { ButtonBorderAnimation } from './ButtonAnimation'
import { ButtonProps, ButtonVariant } from './Button.types'
import { ButtonBase, LeftIcon, RightIcon } from './Button.styled'

function Button({
  variant = ButtonVariant.Filled,
  children,
  leftIcon = null,
  rightIcon = null,
  ...props
}: ButtonProps): JSX.Element {
  return (
    <ButtonBase variant={variant} {...props}>
      {variant !== ButtonVariant.Ghost ? (
        <ButtonBorderAnimation {...props} />
      ) : null}
      {leftIcon ? <LeftIcon>{leftIcon}</LeftIcon> : null}
      {children}
      {rightIcon ? <RightIcon>{rightIcon}</RightIcon> : null}
    </ButtonBase>
  )
}

export default Button

Button Types

export interface ButtonProps {
  children: React.ReactNode
  variant?: 'filled' | 'outlined' | 'ghost'
  size?: 'small' | 'regular'
  underlinedOnHover?: boolean
  leftIcon?: React.ReactNode
  rightIcon?: React.ReactNode
  inverse?: boolean
}

export enum ButtonVariant {
  Filled = 'filled',
  Outlined = 'outlined',
  Ghost = 'ghost',
}

export enum ButtonSize {
  Small = 'small',
  Regular = 'regular',
}

Link Component

import * as React from 'react'
import Button from '../Button/Button'
import { Link as LinkRouter } from 'react-router-dom'
import { LinkProps } from './Link.types'

function Link({ to, ...props }: LinkProps): JSX.Element {
  return <Button to={to} as={LinkRouter} {...props} />
}

export default Link

Link Types

import { ButtonProps } from '../Button/Button.types'
import { LinkProps } from 'react-router-dom'

type RouterLinkWithButtonProps = ButtonProps & LinkProps

export interface LinkProps extends RouterLinkWithButtonProps {}

When I do the above, this problem comes us...

Property 'to' does not exist on type 'IntrinsicAttributes & ButtonProps'.

...which makes sense because button doesn't have the prop to which is required for the Link component from react-router-dom.

How will you approach something like this? Where when using the Button the to prop shouldn't even be in the type and when using the Link the to should be required.

Uses

<Button>Hello</Button>
<Link to="/somewhere">Hello</Link>

from Using styled components "as" prop with typescript